1. Field of the Invention
The present invention relates to an image processing device and an image processing method.
Priority is claimed on Japanese Patent Application No. 2011-230674, filed Oct. 20, 2011, the content of which is incorporated herein by reference.
2. Description of the Related Art
All patents, patent applications, patent publications, scientific articles, and the like, which will hereinafter be cited or identified in the present application, will hereby be incorporated by reference in their entirety in order to describe more fully the state of the art to which the present invention pertains.
In an image processing device, which is included in an imaging apparatus such as a still image camera, a moving image camera, a medical endoscope camera, or an industrial endoscope camera, image processing is performed multiple times in order to generate one still image. Furthermore, as a processing method when performing each image processing, a method of dividing an image of one frame into rectangular blocks and performing each image processing in units of the divided blocks has been proposed (refer to Japanese Patent Publication No. 4179701, hereinafter referred to as Patent Document 1).
Meanwhile, it has been generally known that image data output from a solid-state imaging device such as an imager included in a camera includes distortion due to an optical system of a camera such as magnification chromatic aberration or distortion aberration. Therefore, an image processing device for performing image processing for correcting distortion of an optical system such as magnification chromatic aberration or distortion aberration has been proposed (refer to Japanese Unexamined Patent Application, First Publication No. 2005-44098, hereinafter referred to as Patent Document 2). The image processing device in the related art disclosed in Patent Document 2 collectively sets correction of the magnification chromatic aberration and correction of the distortion aberration as distortion correction, and simultaneously performs image processing using the same distortion correction processing unit.
FIG. 7 is a block diagram illustrating a schematic configuration of the image processing device in accordance with the related art disclosed in Patent Document 2. The image processing device in accordance with the related art illustrated in FIG. 7 has a pipeline in which each image processing necessary for generating one still image is directly interconnected. In more detail, the image processing device in the related art has a pipeline for performing image processing of respective blocks in sequence of a DRAM (Dynamic Random Access Memory), an input DMA (Direct Memory Access) unit, a YC (luminance color conversion) processing unit, an NR (noise removal) processing unit, a distortion correction processing unit, an output DMA unit, and the DRAM. In the image processing device in accordance with the related art illustrated in FIG. 7, the setting or operation of each image processing unit constituting the pipeline is controlled by a sequencer that controls all of the pipeline processing.
Each image processing is configured by the pipeline as illustrated in FIG. 7, resulting in the suppression of pressure to a bus band of the DRAM due to the image processing for generating one still image. Consequently, it is possible to perform the image processing for generating one still image in a short time, that is, to perform the processing at a high speed.
Meanwhile, when a distortion correction process is performed in units of blocks, image data in a range larger than a range (a block) of an image to be output needs to be input to a distortion correction processing unit. Moreover, the input range of the image data to be input to the distortion correction processing unit is changed based on blocks of an image to be processed for output. This is because the appearance of distortion due to an optical system of a camera is changed based on the positions of blocks obtained by dividing an image of one frame. In this regard, the image processing device in the related art disclosed in Patent Document 2 is included with a distortion correction range calculation unit for calculating in advance an input range of image data required when correcting distortion of blocks of an image to be output. That is, the image processing device in the related art disclosed in Patent Document 2 is configured to perform a calculation process of the input range of image data, which requires a lot of processing time, using a dedicated element.
Before the pipeline processing is started, the sequencer allows the distortion correction range calculation unit to calculate in advance the input range of image data, which is required when performing the distortion correction process, in each block to be output. The distortion correction range calculation unit calculates the input area of image data before the distortion correction process (that is, in a distorted state), which corresponds to the output range of image data to be output through the distortion correction process, and calculates a rectangular range, which is circumscribed about (surrounds) the calculated input area, as the input range of image data required in order to obtain the output range of image data to be output through the distortion correction process.
FIG. 8 is a block diagram illustrating a schematic configuration of the distortion correction range calculation unit included in the image processing device in accordance with the related art disclosed in Patent Document 2. As illustrated in FIG. 8, the distortion correction range calculation unit included in the image processing device in accordance with the related art includes a distortion correction coordinate conversion coefficient setting register, a distortion correction output range setting register, a distortion correction range calculation block, and a distortion correction input range result register. When the sequencer allows the distortion correction range calculation unit to calculate the input range of image data at the time of performing of a distortion correction process, the sequencer sets a distortion correction coordinate conversion coefficient and a distortion correction output range in the distortion correction range calculation unit, and allows the distortion correction range calculation unit to calculate the input range of image data.
The distortion correction range calculation block employs the distortion correction output range set by the sequencer as a block to be processed (the output range of image data to be output through a distortion correction process). Then, the distortion correction range calculation block calculates the positions (coordinates) of pixels in image data before performing the distortion correction process, which correspond to respective pixels included in image data in the distortion correction output range, based on the distortion correction coordinate conversion coefficient set by the sequencer, that is, the distortion correction range calculation block performs coordinate conversion. In this way, the input area (after the coordinate conversion) in a distorted state as illustrated in FIG. 7 is obtained. Then, the distortion correction range calculation block calculates a distortion correction input range, which is obtained by setting a rectangular range circumscribed about (surrounding) the input area obtained through the coordinate conversion as the input range of image data required in order to obtain the output range (the distortion correction output range) of image data to be output through the distortion correction process, and stores the calculated distortion correction input range in the distortion correction input range result register. Then, the distortion correction range calculation block outputs a completion interrupt to the sequencer, wherein the completion interrupt represents that the calculation of the input range of the image data has been completed.
If the completion interrupt representing that the calculation of the input range of the image data has been completed is received from the distortion correction range calculation unit, the sequencer starts the pipeline processing. When starting the pipeline processing, the sequencer performs setting required for each image processing unit constituting the pipeline, and then allows the operation of each image processing unit to be performed. At this time, the sequencer sets the distortion correction input range, which is stored in the distortion correction input range result register, in the distortion correction processing unit as the input range of the image data when performing the distortion correction process.
Meanwhile, it has been generally known that, when the magnification chromatic aberration correction is performed based on RAW data (Bayer data in FIG. 7) output from an imager, rather than based on three-plate image data (YCbCr data in FIG. 7) through a YC process or image data (YCbCr data in FIG. 7) subject to an NR process, it is possible to correct the magnification chromatic aberration more accurately, thereby generating a high quality image. The reason for this is that, in relation to the magnification chromatic aberration, since a shift amount of object light due to an optical system is changed based on colors (R, G, and B) of pixels included in the imager, it is not possible to accurately correct the shift amount of each color in three-plate image data in which the colors (R, G, and B) have been mixed.
In this regard, dividing the distortion correction process into magnification chromatic aberration correction and distortion aberration correction, which are then performed, while maintaining the same pipeline structure as the image processing device in the related art is considered. That is, configuring a pipeline for performing image processing of respective blocks in sequence of a DRAM, an input DMA unit, a magnification chromatic aberration correction processing unit, a YC processing unit, an NR processing unit, a distortion aberration correction processing unit, an output DMA unit, and the DRAM may be considered. The pipeline is configured as above so that it is possible to generate a high quality image.
However, when the magnification chromatic aberration correction is configured separately from the distortion aberration correction in one pipeline, the distortion correction range calculation unit included in the image processing device in accordance with the related art as illustrated in FIG. 8 may not accurately perform the calculation of input ranges suitable for the magnification chromatic aberration correction and the distortion aberration correction. This is because the YC process and the NR process are performed between a magnification chromatic aberration correction process and a distortion aberration correction process. In more detail, as illustrated in FIG. 8, the distortion correction range calculation unit included in the image processing device in accordance with the related art includes only one pair of the distortion correction coordinate conversion coefficient setting register and the distortion correction output range setting register, and only one distortion correction input range result register. Therefore, the distortion correction range calculation unit in the related art performs only a calculation process of one type of input range through the activation of a one-time calculation operation of an input range of image data. That is, the distortion correction range calculation unit in the related art performs only one calculation process of the calculation of the input range of image data for the magnification chromatic aberration correction and the calculation of the input range of image data for the distortion aberration correction through the activation of the one-time calculation operation of the input range of image data. Although the calculation of the input ranges of image data corresponding to the magnification chromatic aberration correction and the distortion aberration correction has been performed through the activation of the one-time calculation operation, input ranges with a margin, that is, ranges larger than input ranges suitable for the magnification chromatic aberration correction and the distortion aberration correction, may be calculated as input ranges in consideration of the YC process and the NR process which are performed between the magnification chromatic aberration correction process and the distortion aberration correction process.
Therefore, it is necessary for the distortion correction range calculation unit in the related art to perform a calculation operation of the input range of image data for the distortion aberration correction separately from a calculation operation of the input range of image data for the magnification chromatic aberration correction. However, in this case, loss time occurs in a processing time of the pipeline processing.
FIG. 9 is a timing chart illustrating timings when the distortion correction process in pipeline processing in the image processing device in accordance with the related art is divided into the magnification chromatic aberration correction process and the distortion aberration correction process and performed. In addition, the timing chart illustrated in FIG. 9 illustrates timings when the setting of the image processing units (the magnification chromatic aberration correction processing unit, the YC processing unit, the NR processing unit, and the distortion aberration correction processing unit) is performed in order to perform pipeline processing of a block N+1 while pipeline processing of a block N is being performed.
If the pipeline processing of the block N is started, the sequencer initially sets an output range of image data of a block to be output through the distortion aberration correction process as a distortion correction output range, further sets a distortion correction coordinate conversion coefficient corresponding to the distortion aberration correction process, and allows the distortion correction range calculation unit in the related art to perform the calculation operation of the input range of image data for the distortion aberration correction in a process A. Thereby, the distortion correction range calculation unit in the related art performs the calculation operation a of the input range of the image data for the distortion aberration correction. Then, if the calculation operation a of the input range of the image data for the distortion aberration correction is completed, the distortion correction range calculation unit in the related art outputs a completion interrupt to the sequencer, wherein the completion interrupt represents that the calculation of the input range of the image data has been completed.
If the completion interrupt representing that the calculation of the input range of the image data has been completed is received from the distortion correction range calculation unit, the sequencer sets the input range (a distortion correction input range) of the image data for the distortion aberration correction calculated by the distortion correction range calculation unit in the distortion aberration correction unit in a process B.
Then, in a process C, the sequencer calculates the input range of image data required for the NR processing unit based on the distortion correction input range calculated by the distortion correction range calculation unit, and performs setting required for the NR processing unit based on the calculated input range of the image data. Then, in a process D, the sequencer calculates the input range of image data required for the YC processing unit based on the calculated input range of the image data required for the NR processing unit, and performs setting required for the YC processing unit based on the calculated input range of the image data.
Then, in a process E, the sequencer sets the calculated input range of the image data required for the YC processing unit to the distortion correction output range as the output range of the image data of the block to be output through the magnification chromatic aberration correction process, further sets a distortion correction coordinate conversion coefficient corresponding to the magnification chromatic aberration correction process, and allows the distortion correction range calculation unit in the related art to perform the calculation operation of the input range of image data for the magnification chromatic aberration correction. Thereby, the distortion correction range calculation unit in the related art performs a calculation operation e of the input range of the image data for the magnification chromatic aberration correction. Then, if the calculation operation e of the input range of the image data for the magnification chromatic aberration correction is completed, the distortion correction range calculation unit in the related art outputs a completion interrupt to the sequencer, wherein the completion interrupt represents that the calculation of the input range of the image data has been completed.
If the completion interrupt representing that the calculation of the input range of the image data has been completed is received from the distortion correction range calculation unit, the sequencer sets, in a magnification chromatic aberration correction unit, the input range (a distortion correction input range) of the image data for the magnification chromatic aberration correction calculated by the distortion correction range calculation unit in a process F. Consequently, it is possible to start the pipeline processing of the next block N+1.
In this way, the sequencer performs in advance the calculation and setting operations of the input range of image data when performing the pipeline processing of the next block N+1. Then, the sequencer performs an operation for performing the pipeline processing of a next block N+2 in the same manner.
However, as apparent from FIG. 9, while the distortion correction range calculation unit is performing the calculation operation of the input range of image data, since the sequencer waits for the calculation result of the distortion correction range calculation unit, the sequencer performs no operations, resulting in a delay of a timing at which the sequencer performs setting required for the NR processing unit or setting required for the YC processing unit.
As described above, the process in which the distortion correction range calculation unit calculates the input range of image data required for the distortion correction process is a process requiring much time. Therefore, while performing pipeline processing of a block which is currently processed, the calculation of the input range of image data required for each image processing unit for performing the pipeline processing of the next block or the setting required for each image processing unit based on the calculated input range of the image data is likely not to be completed by the sequencer. That is, as illustrated in FIG. 9, the calculation period of the input range of image data of a block (the block N+1) which is to be processed next is likely to be longer than the period of the pipeline processing of a block (the block N) which is currently processed. If the calculation period of the input range of the image data of the block which is to be processed next is longer than the period of the pipeline processing of the block which is currently processed, that is, if the process of the sequencer is not completed while the pipeline processing of the block currently processed is being performed, loss time in processing time may occur between the pipeline processing of the block N and the pipeline processing of the block N+1 as illustrated in FIG. 9. In addition, the processing time to be lost is considered to be increased as a block is subdivided.
In this regard, a configuration in which the magnification chromatic aberration correction process is separated from the pipeline, and the pipeline processing including the distortion aberration correction is performed after the magnification chromatic aberration correction process of the RAW data (the Bayer data) output from the imager is performed has been considered. In this case, the Bayer data stored in the DRAM is first read, and subject to the magnification chromatic aberration correction process, and then the Bayer data after the magnification chromatic aberration correction process is written (stored) in the DRAM again. Thereafter, the Bayer data after the magnification chromatic aberration correction process stored in the DRAM is read, and subject to the image processing (the pipeline processing) including the distortion aberration correction, and then image data (YCbCr data) after the image processing including the distortion aberration correction is written (stored) in the DRAM. With such a configuration, there is considered to be no problem regarding the calculation operation of the input range of image data by the distortion correction range calculation unit as described above, that is, the problem regarding the loss time in processing time which is caused by the relation between the calculation period of the input range of image data of the block which is to be processed next and the period of the pipeline processing of the block which is currently processed.
However, in this case, the image processing for generating one still image is performed through two pipeline processing. In more detail, one still image is generated through the two pipeline processing, that is, the pipeline processing in sequence of the DRAM, the input DMA unit, the magnification chromatic aberration correction processing unit, the output DMA unit, and the DRAM, and the pipeline processing in sequence of the DRAM, the input DMA unit, the YC processing unit, the NR processing unit, the distortion aberration correction processing unit, the output DMA unit, and the DRAM. Therefore, the number of accesses to the DRAM regarding the image processing is increased twice, resulting in an increase of pressure to the bus band of the DRAM required for the image processing. Moreover, loss time in processing time is also likely to occur between the two pipeline processing.